A shellcode is a small piece of code that takes advantage of a software vulnerability to attack a computer. Many shellcodes work by exploiting stack or buffer overflows. Various known techniques can be used to protect against shellcodes exploiting overflows, such as utilizing bounds checking to detect overflows in programs. Current versions of Microsoft Windows® include a security feature called Data Execution Prevention (DEP), which prevents execution of code from non-executable regions (e.g., the stack, the heap), thereby protecting against the exploitation of buffer overflows.
A new type of advanced shellcode, referred to as a “return oriented shellcode,” exploits vulnerabilities in programs not by executing from the stack or heap, but by jumping between locations within the executable image itself. Because a return oriented shellcode runs from executable regions, it completely bypasses protection mechanisms such as DEP that detect buffer overflow exploits.
A control flow graph (CFG) is a graphical representation of all possible execution paths for a program. Each node in the graph represents a piece of code with one or more jump targets, and one or more jumps. Directed edges are used to represent jumps in the control flow. The entry point is represented by a single entry block, through which control flow enters into the graph. The exit block represents the code through which all control flow exits the program.
CFG based enforcement technologies exist, which rely on tracing control transfer instructions. These approaches are useful in certain contexts. However, return oriented shellcodes can bypass execution of control transfer instructions and/or API entry and exit points. In such cases, current CFG based security agents are not able to detect or protect against return oriented shellcodes.
It would be desirable to be able to address these security vulnerabilities.